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An  Ethical  Basis  for  Autonomous  System  Deployment 

Proposal 50397-CI 

Ronald  C.  Arkin,  College  of  Computing,  Georgia  Tech 
FINAL  REPORT  ATTACHMENT 


I.  PROBLEM  STATEMENT 

The  objectives  of  this  project  were  twofold: 

•  To  gauge  opinion  on  the  use  of  lethality  by  autonomous  systems  in  the  battlefield 
from  various  demographic  groups:  the  military,  robotics  researchers,  policy  makers, 
and  the  general  public. 

•  To  embed  an  ethical  code  within  a  robotic  controller  to  govern  its  behavior  in  a 
manner  consistent  with  the  laws  of  war,  rules  of  engagement  and  code  of  conduct  of 
the  military. 

To  accomplish  these  objectives,  the  project  investigated  and  implemented  an  ethical  basis 
for  deployment  of  lethality  in  autonomous  robotic  systems.  Two  main  thrusts  were 
explored. 

1.  The  first  task  addressed  the  ethical  dimensions  of  robotic  weaponry  in  two 
contexts:  the  robot  as  an  extension  of  the  warfighter  and  the  robot  as  an 
autonomous  agent  acting  on  behalf  of  the  warfighter.  A  formal  survey  has  been 
completed  among  a  broad  population  of  relevant  parties  including  military 
personnel,  the  public,  policymakers,  and  roboticists.  The  results  characterize  the 
decision-making  space  for  the  deployment  of  intelligent  robotic  weaponry, 
whereby  the  military  can  judiciously  determine  its  most  effective  and  appropriate 
usage. 

2.  The  second  task  involved  the  generation  of  an  artificial  "conscience"  for  an 
intelligent  autonomous  robotic  agent,  which  applies  limits  and  constraints  on  its 
actions  as  required  by  the  bounds  of  ethical  decision-making.  These  limits  are 
generated  from  the  Laws  of  War,  rules  of  engagement,  and  other  requirements. 
The  intent  is  to  yield  robots  that  can  perhaps  act  more  humanely  than  humans  do 
under  highly  stressful  conditions;  provide  warnings  in  the  field  to  military 
decision-makers  about  the  potential  ethical  consequences  of  tactical  actions 
regarding  the  use  of  this  technology;  and  to  ensure  that  accountability  is 
engineered  into  these  systems  from  the  onset. 


II.  ACCOMPLISHMENT  SUMMMARY 


We  have  successfully  completed  all  of  the  goals  of  this  three-year  project. 

1.  The  survey  was  closed  and  completed  on  October  20,  2007.  The  results  of  the 
completed  survey  are  documented  in  a  technical  report  entitled  Lethality  and 
Autonomous  Systems:  Survey  Design  and  Results,  L.  Moshkina  and  R.  Arkin  2007 
(GIT-GVU-TR-07-16)  and  a  new  book  published  in  May  2009  (Objective  1).  This 
report  summarizes  the  year  one  results  obtained  on  our  survey.  Specifically  it  reports 
the  methods  and  results  of  an  on-line  survey  addressing  the  issues  surrounding 
lethality  and  autonomous  systems.  The  data  from  this  survey  were  analyzed  both 
qualitatively,  providing  a  comparison  between  four  different  demographic  samples 
targeted  in  the  survey  (namely,  robotics  researchers,  policymakers,  the  military,  and 
the  general  public),  and  quantitatively,  for  the  robotics  researcher  demographic.  In 
addition  to  the  analysis,  the  design  and  administration  of  this  survey  and  a  discussion 
of  the  survey  results  are  provided. 

2.  The  design,  development  and  implementation  of  the  ethical  architecture  (Objective 
2),  including  its  philosophy  and  motivation,  formal  mathematical  development,  and 
testing  have  been  completed.  This  is  summarized  in  a  second  technical  report  entitled: 
Governing  Lethal  Behavior:  Embedding  Ethics  in  a  Hybrid  Deliberative/Reactive 
Robot  Architecture,  Arkin,  R.,  2007  (GIT-GVU-TR-07-11)  and  a  new  book  published 
in  May  2009.  This  report  describes  the  basis,  motivation,  theory,  and  design 
recommendations  for  the  implementation  of  an  ethical  control  and  reasoning  system 
potentially  suitable  for  constraining  lethal  actions  in  an  autonomous  robotic  system  so 
that  they  fall  within  the  bounds  prescribed  by  the  Laws  of  War  and  Rules  of 
Engagement.  It  is  based  upon  extensions  to  existing  deliberative/reactive  autonomous 
robotic  architectures,  and  includes  recommendations  for  (1)  post  facto  suppression  of 
unethical  behavior,  (2)  behavioral  design  that  incorporates  ethical  constraints  from 
the  onset,  (3)  the  use  of  affective  functions  as  an  adaptive  component  in  the  event  of 
unethical  action,  and  (4)  a  mechanism  in  support  of  identifying  and  advising 
operators  regarding  the  ultimate  responsibility  for  the  deployment  of  such  a  system. 


II.l  Task  1:  Questionnaire  Summary  (from  Technical  Report) 

A  high-level  summary  of  the  survey  results  follows.  The  interested  reader  should  refer  to 
the  technical  report  mentioned  on  the  previous  page.  After  analyzing  the  results  of  the 
survey,  the  following  generalizations  can  be  made: 

1.  Demographics: 

•  A  typical  respondent  was  an  American  or  Western  European  male  in  his  20s 
or  30s,  with  higher  education,  significant  computer  experience,  and  positive 
attitude  toward  technology  and  robots. 

•  The  participants  ranged  from  under  21  to  over  66  years  old  (all  the 
participants  were  over  18);  11%  of  the  participants  were  female;  non-US 
participants  were  from  all  over  the  world,  including  Australia,  Asia,  Eastern 
Europe  and  Africa. 

2.  Levels  of  Autonomy: 

•  In  general,  regardless  of  roles  or  situations,  the  more  the  control  shifts  away 
from  the  human,  the  less  such  an  entity  is  acceptable  to  the  participants.  A 
human  soldier  was  the  most  acceptable  entity  in  warfare,  followed  by  the 
robot  as  an  extension  of  the  warfighter,  and  autonomous  robot  was  the  least 
acceptable. 

•  There  was  a  larger  gap  in  terms  of  acceptability  between  a  robot  as  an 
extension  and  autonomous  robot  than  that  between  soldier  and  robot  as  an 
extension. 

•  Taking  human  life  by  an  autonomous  robot  in  both  Open  Warfare  and  Covert 
Operations  is  unacceptable  to  more  than  half  of  the  participants  (56% 
disagreed  or  strongly  disagreed),  especially  in  the  case  of  Covert  Operations 
on  Home  Territory. 

3.  Comparison  between  Community  Types: 

•  Regardless  of  roles  or  situations,  in  most  cases  the  general  public  found  the 
employment  of  soldiers  and  robots  less  acceptable  than  any  other  community 
type,  and,  conversely,  those  with  military  experience  and  policymakers  found 
such  employment  more  acceptable. 

•  More  military  and  policymakers  were  in  favor  of  the  same  ethical  standards 
for  both  soldiers  and  robots  than  both  the  general  public  and  roboticists,  who 
were  more  in  favor  of  higher  standards  for  robots. 

•  When  asked  about  the  responsibility  for  any  lethal  errors,  those  with  military 
experience  attributed  the  least  amount  of  blame  to  any  of  the  responsible 
parties. 

4.  Roles: 

•  The  most  acceptable  role  for  using  both  types  of  robots  is  Reconnaissance;  the 
least  acceptable  is  Crowd  Control. 

•  Robots  could  be  used  for  roles  where  less  force  is  involved,  such  as  Sentry 
and  Reconnaissance,  and  should  be  avoided  for  roles  where  use  of  force  may 


be  necessary,  especially  when  civilian  lives  are  at  stake,  such  as  Crowd 
Control  and  Hostage  Rescue. 


5.  Situations: 

•  Covert  Operations  were  less  acceptable  to  the  entire  set  of  participants  than 
Open  Warfare  (whether  on  Home  or  Foreign  Territory). 

6.  Ethical  Considerations: 

•  The  majority  of  participants,  regardless  of  the  community  type,  agreed  that  the 
ethical  standards,  namely,  Laws  of  War,  Rules  of  Engagement,  Code  of 
Conduct  and  Additional  Moral  Standards,  do  apply  to  both  soldiers  (84%)  and 
robots  (72%). 

•  The  more  concrete,  specific  and  identifiable  ethical  standards  were,  the  more 
likely  they  were  to  be  considered  applicable  to  both  soldiers  and  robots,  with 
Laws  of  War  being  the  most  applicable,  and  Additional  Moral  Standards  the 
least. 

•  66%  of  the  participants  were  in  favor  of  higher  ethical  standards  for  a  robot 
than  those  for  a  soldier. 

•  59%  of  the  participants  believed  that  an  autonomous  robot  should  have  a  right 
to  refuse  an  order  it  finds  unethical,  thus  in  a  sense  admitting  that  it  may  be 
more  important  for  a  robot  to  behave  ethically  than  to  stay  under  the  control  of 
a  human. 

7.  Responsibility: 

•  A  soldier  was  the  party  considered  the  most  responsible  for  both  his/her  own 
lethal  errors,  and  for  those  of  a  robot  as  an  extension  under  his/her  control. 
Robots  were  the  least  blamed  parties,  although  an  autonomous  robot  was 
found  blameworthy  twice  as  much  as  robot  as  an  extension.  It  is  interesting 
that  even  though  robots  were  blamed  the  least,  40%  of  the  respondents  still 
found  an  autonomous  robot  responsible  for  its  errors  to  a  very  significant  or 
significant  extent. 

•  As  the  control  shifts  away  from  the  soldier,  the  robot  and  its  maker  should 
take  more  responsibility  for  robot’s  actions.  A  robot  designer  was  blamed 
31%  less  for  the  mistakes  of  a  robot  as  an  extension  than  those  of  an 
autonomous  robot. 

8.  Benefits  and  Concerns: 

•  Saving  lives  of  soldiers  was  considered  the  most  clear-cut  benefit  of 
employing  robots  in  warfare  and  the  main  concern  was  that  of  risking  civilian 
lives.  Saving  soldiers’  lives  and  decreasing  psychological  trauma  to  soldiers 
outweigh  the  risk  to  the  soldiers  the  most.  Decreasing  cost  and  producing 
better  battlefield  outcome  were  also  viewed  as  benefits  rather  than  concerns. 

•  For  the  roboticists,  the  categories  regarding  battlefield  outcomes  and  friendly 
fire  were  not  considered  strongly  as  either  benefits  or  concerns,  suggesting 
that  the  participants  did  not  think  that  robots  would  have  an  effect  on  these 
categories. 


9.  Wars  and  Emotions: 

•  The  majority  of  the  participants  (69%)  believe  that  it  would  be  easier  to  start 
wars  if  robots  were  employed  in  warfare. 

•  Sympathy  was  considered  to  be  beneficial  to  a  military  robot  by  over  half  of 
the  participants  (59%),  and  guilt  by  just  under  a  half  (49%).  The  majority  of 
the  participants  (75%)  were  against  anger  in  a  military  robot. 

10.  Cultural  Background: 

US  participants  were  more  likely  to  accept  both  soldiers  and  robots  in 
proposed  roles  and  situations  than  non-US  participants.  They  favored  less 
stringent  ethical  standards  for  robots  and  were  less  likely  to  give  the  robot  a 
right  to  refuse  an  unethical  order  than  non-US  participants.  They  were  also 
less  likely  to  assign  responsibility  for  lethal  errors  of  soldiers  and  robots  and 
less  willing  to  provide  military  robots  with  emotions. 

11.  Firearms  Experience: 

•  Those  with  less  firearms  experience  found  the  use  of  all  three  levels  of 
autonomy  for  the  proposed  roles,  overall,  less  acceptable  than  those  with  more 
experience,  and  found  the  use  of  both  types  of  robots  less  acceptable  in  the 
proposed  situations. 

•  Those  with  less  firearm  experience  were  also  more  likely  to  hold  a  robot  to 
more  stringent  ethical  standards  when  compared  to  those  of  a  soldier;  more 
likely  to  allow  a  robot  to  refuse  an  unethical  order,  more  prone  to  assign 
responsibility  for  lethal  errors  of  soldier  and  robot  as  extension,  and  more 
willing  to  provide  military  robots  with  the  emotions  of  sympathy,  guilt  and 
happiness. 

12.  Spirituality: 

In  most  cases,  spirituality  had  no  effect  on  the  participants’  opinions  with  the 
exception  of  the  use  of  robot  as  an  extension  for  the  proposed  roles  and  the 
use  of  all  three  levels  of  autonomy  in  the  given  situations.  Those  of  higher 
spirituality  found  such  use  more  acceptable  in  warfare;  also,  more 
spiritual/religious  participants  were  less  convinced  that  it  would  be  easier  to 
start  wars  if  robots  were  brought  onto  the  battlefield. 


II.  2  Task  2:  Ethical  Architecture  Summary 

The  process  for  conducting  Task  2  involved  the  following  stages: 

1.  Understanding  of  philosophical  and  legal  underpinnings: 

We  first  presented  the  background,  motivation  and  philosophy  for  the  design 
of  an  ethical  autonomous  robotic  system  capable  of  using  lethal  force.  The 
system  is  governed  by  the  Laws  of  War  and  Rules  of  Engagement  using  them 
as  constraints. 

Arkin,  R.C.,  "Governing  Lethal  Behavior:  Embedding  Ethics  in  a  Hybrid  Deliberative/Reactive 
Robot  Architecture  -  Part  I:  Motivation  and  Philosophy",  Proc.  Human-Robot  Interaction  2008 , 
Amsterdam,  NL,  March  2008. 

2.  Development  of  underlying  mathematical  formalisms: 

We  provided  the  permeating  formalisms  for  a  hybrid  deliberative/reactive 
architecture  designed  to  govern  the  application  of  lethal  force  by  an  autonomous 
system  to  ensure  that  it  conforms  with  International  Law. 

Arkin,  R.C.,  "Governing  Lethal  Behavior:  Embedding  Ethics  in  a  Hybrid  Deliberative/Reactive 
Robot  Architecture  -  Part  II:  Formalization  for  Ethical  Control",  Proc.  1st  Conference  on  Artificial 
General  Intelligence,  Memphis,  TN,  March  2008. 

3.  Design  of  architectural  principles: 

We  then  provided  the  representational  requirements,  architectural  design  criteria 
and  recommendations  to  design  and  construct  an  autonomous  robotic  system 
architecture  capable  of  the  ethical  use  of  lethal  force. 

Arkin,  R.C.,  "Governing  Lethal  Behavior:  Embedding  Ethics  in  a  Hybrid  Deliberative/Reactive 
Robot  Architecture  -  Part  III:  Representational  and  Architectural  Considerations",  Proceedings  of 
Technology  in  Wartime  Conference,  Palo  Alto,  CA,  January  2008. 

4.  Implementation  of  ethical  governor: 

In  order  to  evaluate  the  feasibility  of  the  ethical  governor  implementation,  a  series 
of  test  scenarios  were  developed  within  the  MissionLab  simulation  environment. 
A  variety  of  situations  were  presented  to  an  autonomous  fixed-wing  UAV  in 
which  the  ethical  use  of  lethal  force  must  be  ensured. 

Arkin,  R.C.,  Ulam,  P.,  and  Duncan,  B.,  “An  Ethical  Governor  for  Constraining  Lethal  Action  in 
an  Autonomous  System",  GVU  Technical  Report  GIT-GVU-09-02,  GVU  Center,  Georgia 
Institute  of  Technology,  2009. 

5.  Implementation  of  responsibility  advisor: 

An  ethical  permission  responsibility  advisor  was  prototyped  and  demonstrated  in 
a  manner  fully  consistent  with  the  overarching  architectural  principles  developed 
earlier. 

Arkin,  R.C.,  Wagner,  A.,  and  Duncan,  B.,  "Responsibility  and  Lethality  for  Unmanned  Systems: 
Ethical  Pre-mission  Responsibility  Advisement",  Proc.  2009  IEEE  Workshop  on  Roboethics, 


Kobe  JP,  May  2009. 

6.  Implementation  of  moral  emotions: 

Using  a  cognitive  model  of  guilt  we  have  implemented  it  computationally  and 
created  a  proof  of  concept  demonstration  in  a  military  context,  demonstrating  its 
utility  for  altering  behavior  based  on  emotional  state. 

Arkin,  R.C.,  and  Ulam,  P.,  “An  Ethical  Adaptor:  Behavioral  Modification  Derived  from  Moral 
Emotions",  GVU  Technical  Report  GIT-GVU-09-04,  GVU  Center,  Georgia  Institute  of 
Technology,  2009. 

Suitable  ethical  test  scenarios  have  been  completed  and  implemented  as  a  prototype 
within  MissionLab,  our  laboratory’s  mission  specification  software  system.  Numerous 
videos  document  these  results: 

•  Operator  interface  for  the  ethical  governor: 
ftp://ftp.cc.gatech.edu/pub/groups/robot/videos/PTF  Interface  Final  Largev3.mpg 

•  Demonstration  of  the  Ethical  Responsibility  Advisor: 
http://www.cc.gatech.edu/ai/robot-lab/ethics/res-advisor.mpg 

•  Demonstration  of  the  Ethical  Governor: 
ftp://ftp.cc.gatech.edu/pub/groups/robot/videos/ethics  governor  final  largev3.mpg 

•  Demonstration  of  the  Ethical  Adaptor  (Guilt  mechanism) 
ftp://ftp.cc.gatech.edu/pub/groups/robot/videos/guilt  movie  v3.mpg 


The  full  discourse  on  this  subject  is  available  in 


Arkin,  R.C.,  Governing  Lethal  Behavior  in  Autonomous  Systems,  Chapman-Hall,  2009. 


